{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bitkira/Colab/blob/main/tutorial_notebooks_zh/aflow_optimizer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install git+https://github.com/EvoAgentX/EvoAgentX.git"
      ],
      "metadata": {
        "id": "Ly0PXMWDqaNq"
      },
      "id": "Ly0PXMWDqaNq",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install PyPDF2 selenium html2text fastmcp"
      ],
      "metadata": {
        "id": "r0tVxsVGucb2"
      },
      "id": "r0tVxsVGucb2",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "2d722a07",
      "metadata": {
        "id": "2d722a07"
      },
      "source": [
        "# AFlow 优化器教程\n",
        "\n",
        "本教程将指导你如何使用 EvoAgentX 的 AFlow 优化器来优化你的工作流。AFlow 优化器是一个强大的工具，可以帮助你自动优化工作流的性能。\n",
        "\n",
        "## 1. 概述\n",
        "\n",
        "AFlow 优化器是 EvoAgentX 框架中的一个重要组件，它提供了以下功能：\n",
        "\n",
        "- 自动优化工作流的性能\n",
        "- 支持多种优化策略\n",
        "- 提供详细的优化报告\n",
        "- 支持自定义优化目标\n",
        "\n",
        "## 2. 设置 AFlow 优化器\n",
        "\n",
        "首先，你需要导入相关模块并设置 AFlow 优化器。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ced34112",
      "metadata": {
        "id": "ced34112"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from dotenv import load_dotenv\n",
        "from evoagentx.optimizers import AFlowOptimizer\n",
        "from evoagentx.models import LiteLLMConfig, LiteLLM, OpenAILLMConfig, OpenAILLM\n",
        "from evoagentx.benchmark import AFlowHumanEval\n",
        "\n",
        "try:\n",
        "    from google.colab import userdata\n",
        "    OPENAI_API_KEY = userdata.get(\"OPENAI_API_KEY\")\n",
        "    ANTHROPIC_API_KEY = userdata.get(\"ANTHROPIC_API_KEY\")\n",
        "except ImportError:\n",
        "    OPENAI_API_KEY = None\n",
        "    ANTHROPIC_API_KEY = None\n",
        "\n",
        "if not OPENAI_API_KEY or not ANTHROPIC_API_KEY:\n",
        "    load_dotenv()\n",
        "    OPENAI_API_KEY = OPENAI_API_KEY or os.getenv(\"OPENAI_API_KEY\")\n",
        "    ANTHROPIC_API_KEY = ANTHROPIC_API_KEY or os.getenv(\"ANTHROPIC_API_KEY\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2b80e0ed",
      "metadata": {
        "id": "2b80e0ed"
      },
      "source": [
        "\n",
        "### 配置LLM模型\n",
        "按照原始AFlow实现中的设置，AFlow优化器使用两种不同的LLM：\n",
        "1. 一个优化器LLM（例如Claude 3.5 Sonnet）用于工作流优化\n",
        "2. 一个执行器LLM（例如GPT-4o-mini）用于任务执行"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "93b8ddc6",
      "metadata": {
        "id": "93b8ddc6"
      },
      "outputs": [],
      "source": [
        "# Configure the optimizer LLM (Claude 3.5 Sonnet)\n",
        "claude_config = LiteLLMConfig(\n",
        "    model=\"anthropic/claude-3-5-sonnet-20240620\",\n",
        "    anthropic_key=ANTHROPIC_API_KEY\n",
        ")\n",
        "optimizer_llm = LiteLLM(config=claude_config)\n",
        "\n",
        "# Configure the executor LLM (GPT-4o-mini)\n",
        "openai_config = OpenAILLMConfig(\n",
        "    model=\"gpt-4o-mini\",\n",
        "    openai_key=OPENAI_API_KEY\n",
        ")\n",
        "executor_llm = OpenAILLM(config=openai_config)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "afd49fdb",
      "metadata": {
        "id": "afd49fdb"
      },
      "source": [
        "\n",
        "## 3. 设置组件\n",
        "\n",
        "### 第一步：定义任务配置\n",
        "\n",
        "AFlow 优化器需要一个配置来指定任务类型和可用的操作符。以下是不同任务类型的示例配置：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "45dc12c5",
      "metadata": {
        "id": "45dc12c5"
      },
      "outputs": [],
      "source": [
        "EXPERIMENTAL_CONFIG = {\n",
        "    \"humaneval\": {\n",
        "        \"question_type\": \"code\",\n",
        "        \"operators\": [\"Custom\", \"CustomCodeGenerate\", \"Test\", \"ScEnsemble\"]\n",
        "    },\n",
        "    \"mbpp\": {\n",
        "        \"question_type\": \"code\",\n",
        "        \"operators\": [\"Custom\", \"CustomCodeGenerate\", \"Test\", \"ScEnsemble\"]\n",
        "    },\n",
        "    \"hotpotqa\": {\n",
        "        \"question_type\": \"qa\",\n",
        "        \"operators\": [\"Custom\", \"AnswerGenerate\", \"QAScEnsemble\"]\n",
        "    },\n",
        "    \"gsm8k\": {\n",
        "        \"question_type\": \"math\",\n",
        "        \"operators\": [\"Custom\", \"ScEnsemble\", \"Programmer\"]\n",
        "    },\n",
        "    \"math\": {\n",
        "        \"question_type\": \"math\",\n",
        "        \"operators\": [\"Custom\", \"ScEnsemble\", \"Programmer\"]\n",
        "    }\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a3686dd2",
      "metadata": {
        "id": "a3686dd2"
      },
      "source": [
        "\n",
        "### 第二步：定义初始工作流\n",
        "\n",
        "AFlow 优化器需要两个文件：\n",
        "- `graph.py`：该文件用 Python 代码定义初始工作流图。\n",
        "- `prompt.py`：该文件定义工作流中使用的提示。\n",
        "\n",
        "以下是 HumanEval 基准的 `graph.py` 文件示例：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "868459d6",
      "metadata": {
        "id": "868459d6"
      },
      "outputs": [],
      "source": [
        "import evoagentx.workflow.operators as operator\n",
        "import examples.aflow.code_generation.prompt as prompt_custom # noqa: F401\n",
        "from evoagentx.models.model_configs import LLMConfig\n",
        "from evoagentx.benchmark.benchmark import Benchmark\n",
        "from evoagentx.models.model_utils import create_llm_instance\n",
        "\n",
        "class Workflow:\n",
        "\n",
        "    def __init__(\n",
        "        self,\n",
        "        name: str,\n",
        "        llm_config: LLMConfig,\n",
        "        benchmark: Benchmark\n",
        "    ):\n",
        "        self.name = name\n",
        "        self.llm = create_llm_instance(llm_config)\n",
        "        self.benchmark = benchmark\n",
        "        self.custom = operator.Custom(self.llm)\n",
        "        self.custom_code_generate = operator.CustomCodeGenerate(self.llm)\n",
        "\n",
        "    async def __call__(self, problem: str, entry_point: str):\n",
        "        \"\"\"\n",
        "        工作流的实现\n",
        "        Custom 操作符可以生成任何你想要的内容。\n",
        "        但当你想获取标准代码时，应该使用 custom_code_generate 操作符。\n",
        "        \"\"\"\n",
        "        # await self.custom(input=, instruction=\"\")\n",
        "        solution = await self.custom_code_generate(problem=problem, entry_point=entry_point, instruction=prompt_custom.GENERATE_PYTHON_CODE_PROMPT) # 但当你想获取标准代码时，应该使用 customcodegenerator\n",
        "        return solution['response']"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ff582e54",
      "metadata": {
        "id": "ff582e54"
      },
      "source": [
        "\n",
        "!!! 注意\n",
        "    在定义工作流时，请注意以下关键点：\n",
        "\n",
        "    1. **提示导入路径**：确保正确指定 `prompt.py` 的导入路径（例如 `examples.aflow.code_generation.prompt`）。此路径应该与你的项目结构匹配，以便正确加载提示。\n",
        "\n",
        "    2. **操作符初始化**：在 `__init__` 函数中，你必须初始化工作流中将使用的所有操作符。每个操作符都应该使用适当的 LLM 实例进行实例化。\n",
        "\n",
        "    3. **工作流执行**：`__call__` 函数是工作流执行的主要入口点。它应该定义工作流的完整执行逻辑，并返回将用于评估的最终输出。\n",
        "\n",
        "\n",
        "以下是 HumanEval 基准的 `prompt.py` 文件示例：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "163389d6",
      "metadata": {
        "id": "163389d6"
      },
      "outputs": [],
      "source": [
        "GENERATE_PYTHON_CODE_PROMPT = \"\"\"\n",
        "Generate a functional and correct Python code for the given problem.\n",
        "\n",
        "Problem: \"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e258b56e",
      "metadata": {
        "id": "e258b56e"
      },
      "source": [
        "\n",
        "!!! 注意\n",
        "    如果工作流不需要任何提示，则 `prompt.py` 文件可以为空。\n",
        "\n",
        "### 第三步：准备基准\n",
        "\n",
        "在本教程中，我们将使用 AFlowHumanEval 基准。它遵循与 [原始 AFlow 实现](https://github.com/FoundationAgents/MetaGPT/tree/main/examples/aflow) 中相同的数据划分和格式。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8768c214",
      "metadata": {
        "id": "8768c214"
      },
      "outputs": [],
      "source": [
        "# Initialize the benchmark\n",
        "humaneval = AFlowHumanEval()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b35ea92a",
      "metadata": {
        "id": "b35ea92a"
      },
      "source": [
        "\n",
        "## 4. 配置和运行 AFlow 优化器\n",
        "\n",
        "AFlow 优化器可以通过各种参数进行配置，以控制优化过程：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e3bab9cd",
      "metadata": {
        "id": "e3bab9cd"
      },
      "outputs": [],
      "source": [
        "optimizer = AFlowOptimizer(\n",
        "    graph_path=\"examples/aflow/code_generation\",  # 初始工作流图的路径\n",
        "    optimized_path=\"examples/aflow/humaneval/optimized\",  # 保存优化工作流的路径\n",
        "    optimizer_llm=optimizer_llm,  # 用于优化的 LLM\n",
        "    executor_llm=executor_llm,    # 用于执行的 LLM\n",
        "    validation_rounds=3,          # 优化期间在开发集上运行验证的次数\n",
        "    eval_rounds=3,               # 测试期间在测试集上运行评估的次数\n",
        "    max_rounds=20,               # 最大优化轮数\n",
        "    **EXPERIMENTAL_CONFIG[\"humaneval\"]  # 特定任务的配置，用于指定任务类型和可用操作符\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3e285116",
      "metadata": {
        "id": "3e285116"
      },
      "source": [
        "\n",
        "### 运行优化\n",
        "\n",
        "要开始优化过程：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3676d8fa",
      "metadata": {
        "id": "3676d8fa"
      },
      "outputs": [],
      "source": [
        "# 优化工作流\n",
        "optimizer.optimize(humaneval)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f68c2027",
      "metadata": {
        "id": "f68c2027"
      },
      "source": [
        "\n",
        "!!! 注意\n",
        "    在优化过程中，工作流将在每一步骤上对开发集进行 `validation_rounds` 次验证。确保基准 `humaneval` 包含开发集（即 `self._dev_data` 不为空）。\n",
        "\n",
        "### 测试优化后的工作流\n",
        "\n",
        "要测试优化后的工作流：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e0248df3",
      "metadata": {
        "id": "e0248df3"
      },
      "outputs": [],
      "source": [
        "# 测试优化后的工作流\n",
        "optimizer.test(humaneval)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "87b35d19",
      "metadata": {
        "id": "87b35d19"
      },
      "source": [
        "默认情况下，优化器将选择验证性能最高的工作流进行测试。你还可以使用 `test_rounds: List[int]` 参数指定测试轮次。例如，要评估第二轮和第三轮，可以使用 `optimizer.test(humaneval, test_rounds=[2, 3])`。\n",
        "\n",
        "!!! 注意\n",
        "    在测试期间，工作流将在测试集上进行 `eval_rounds` 次评估。确保基准 `humaneval` 包含测试集（即 `self._test_data` 不为空）。\n",
        "\n",
        "有关完整的工作示例，请参阅 [aflow_humaneval.py](https://github.com/EvoAgentX/EvoAgentX/blob/main/examples/optimization/aflow/aflow_humaneval.py)。"
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "name": "python"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}